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Abstract 

This paper establishes a sharp condition on the restricted isometry property (RIP) for 
both the sparse signal recovery and low-rank matrix recovery. It is shown that if the mea- 
surement matrix A satisfies the RIP condition (5^ < 1/3, then all fc-sparse signals (3 can be 
recovered exactly via the constrained £i minimization based on y = Af3. Similarly, if the 
linear map A4 satisfies the RIP condition (5;^ < 1/3, then all matrices X of rank at most r 
can be recovered exactly via the constrained nuclear norm minimization based on b = A4{X). 
Furthermore, in both cases it is not possible to do so in general when the condition does 
not hold. In addition, noisy cases are considered and oracle inequalities arc given under the 
sharp RIP condition. 

Keywords: Compressed sensing; Dantzig selector; ii minimization; low-rank matrix recovery; 
nuclear norm minimization; restricted isometry; sparse signal recovery. 



1 Introduction 

Compressed sensing has been a very active field of recent research with a wide range of ap- 
plications, including signal processing, medical imaging, seismology, and statistics. The goal 
is to develop efficient data acquisition techniques that allow accurate reconstruction of highly 
undersampled sparse signals. It is now well understood that the constrained li minimization 
*The research of Tony Cai was supported in part by NSF FRG Grant DMS-0854973. 
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method provides an effective way for recovering sparse signals. See, e.g., Candes and Tao [HE], 
Donoho [11] and Donoho, Elad, and Temlyakov [12]. A closely related problem is the affine rank 
minimization problem, where the goal is to recover a large low-rank matrix based on an observa- 
tion of an affine transformation of the matrix. Applications include linear system identification 
and control, Euclidean embedding, and image compression. See, e.g., Candes and Plan [9|, and 
Recht, Fazel and Parrilo |18j . 

More specifically, in compressed sensing, one observes (^4, y) with 

y = Ap + z (1) 

where y G M", A G W^^^ with n <C p, /3 G is a sparse signal of interest, and z G M" is a vector 
of measurement errors. One wishes to recover the unknown sparse signal /3 G based on A 
and y using an efficient algorithm. The affine rank minimization problem aims to reconstruct a 
low-rank matrix X based on a known linear map A4 and an observed vector 6 G M"? where 

b = M{X) + z. (2) 

Here A4 : M'"^" — )• M'' is a linear map, X G M'"^" is an unknown low-rank matrix of interest, 
and z G M'' is an error vector. 

The methods of constrained ii and nuclear norm minimization, 

(Pb) P = argmm{||/3||i : - y g B} (3) 
(Pb) X, = argmin{||X||, : MiX) -beB}, (4) 

as convex relaxations to io and rank minimization respectively, have been shown to be very 
effective in solving these problems. Here ||X||* is the nuclear norm of X , which is defined to be 
the sum of the singular values of X, and ;B is a bounded set determined by the noise structure. 
For example, B = {0} in the noiseless case and B is the feasible set of the error vector z in the 
case of bounded noise. 

One of the most commonly used frameworks for sparse signal and low-rank matrix recovery 
is the Restricted Isometry Property (RIP). See Candes and Tao [6] and Recht et al. jl8j . A 
vector is said to be /c-sparse if |supp(t;)| < k, where supp(f) = {i : Vi ^ 0} is the support of 
V. In this paper, we shall use the phrase "r-rank matrices" to refer to matrices of rank at most 
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r. In compressed sensing, the RIP requires subsets of certain cardinality of the columns of A 
to be close to an orthonormal system. The RIP conditions for the signal and matrix recovery 
are similar and we shall state them together to save space. Let A G M"^^ be a matrix and let 
M : M"^^" — )• M'? be a linear map. For integers 1 < k < p and 1 < r < min{m, n}, define the 
restricted isometry constants (RIC) 5^ and 5^ to be the smallest non-negative numbers such 
that for all /c-sparse vectors /3 and all r-rank matrices X, 

{i-s^m\l < wmi < (5) 

{l-6^)\\X\\l< \\M{xm <{l + 5^)\\X\\l (6) 
where ||^|||' = J2^ij the squared Frobenius norm of X = (xij). 

A major focus of compressed sensing is to find explicit and simple conditions under which 
the sparse signals can be recovered exactly using a computational efficient algorithm. A variety 
of sufficient conditions on the RIP for the exact/stable recovery of /c-sparse signals and r-rank 
matrices have been introduced in the literature. Sufficient conditions for the signal recovery 
include (54 < \/2 - 1 in Candes ^2k < 0-472 in Cai, Wang and Xu < 0.493 in Mo and 

Li [15] and (5^ < 0.307 in Cai, Wang and Xu [1]; for the matrix recovery, sufficient conditions 
are (5^ < \/2- 1 in Candes and Plan [9], 5^ < 0.607, 6^ < 0.558, 5^ < 0.4721 in Mohan and 
Fazel [IS], < 0.4931, < 0.307 in Wang and Li [2D]. On the other hands, negative results 
have also been obtained. In the case of signal recovery, Davies and Gribonval [10] and Cai, 
Wang and Xu [1] showed respectively that it is impossible to recover certain A;-sparse signals 
when > \/2/2 and when 6^ = 2k\ < 0.5. For matrix recovery, Wang and Li [20] proved 
that nuclear norm minimization cannot recover exactly all rank r matrices in the noiseless case 
when (5^ = 1/3 or = + e, where e is arbitrarily small. 

Among those RIP conditions, the ones on 5^ and 6^ are arguably the most natural for the 
reconstruction of fc-sparse signals and r-rank matrices, respectively. The main goal of this paper 
is to establish a sharp condition on 5^ and 5^. Specifically, we show that in the noiseless case 
{z = 0) the conditions 

5^<\ and 5^<\ (7) 

are sharp respectively for the exact recovery of fc-sparse signals based on ([T|) and for the exact 
recovery of r-rank matrices based on Q . These conditions are also sharp for the stable recovery 
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in the noisy case. That is, under the condition 5^ < 1/3, all fc-sparse signals can be exactly 
recovered via the constrained ii minimization ([3]) in the noiseless case and can be stably recovered 
in the noisy case. Furthermore, it is not possible to do so in general if > 1/3. Similarly, 
for the recovery of r-rank matrices using the constrained nuclear norm minimization based on 
([2]), the condition 5^ < 1/3 is sharp. To the best of our knowledge, d?]) is the first sharp RIP 
condition. 

Various oracle inequalities have been given in the literature for the constrained £i /nuclear 
norm minimization estimators, known as the Dantzig Selector, in the setting of Gaussian noise. 
See, for example, Candes and Tao [7j and Cai, Wang and Xu j3] for the sparse signal recovery 
and Candes and Plan [9] for the matrix recovery under the condition 5^ < \/2 — 1. In this 
paper we derive oracle inequalities for both sparse signal and low-rank matrix recovery under 
the condition <l/3 and 5:^ < 1/3. 

Besides providing a sharp condition on 6^ and (5^ , the same techniques can also be used to 
sharpen other RIP conditions such as (5^ and 5^. We show that, in the noiseless case, (5^ < 1/2 
and < 1/2 are respectively sufficient for the exact recovery of fc-sparse signals based on ([1]) 
and for the exact recovery of r-rank matrices based on ([2]). 

The rest of the paper is organized as follows. Section [2] reviews basic notations and defini- 
tions and states some useful facts on the null spaces. Section [3.11 then introduces a technically 
important tool called the Division Lemma, which is used in the detailed analysis for both the 
signal and matrix recovery. Sections 13.21 and 13.31 separately analyze the sparse signal recovery 
and low-rank matrix recovery, in both the noiseless and noisy settings. Section |4] provides oracle 
inequalities for Gaussian noise under the conditions 6^ < 1/3 and 5^ < 1/3, and discusses 
other RIP conditions. The proofs of the main results are given in Section [5j 

2 Notations and Preliminaries 

In this section, we introduce basic notations and definitions that will be used throughout the 
paper, and state some facts on the null spaces that will be used later. 

For a vector v = (fi,--- G W, define fmax(A:) to be the vector v with all but the 
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largest k entries in absolute values set to zero, and let w_max{fc) = v — t^max(fc)- For ^ matrix 
X £ i^m-x" (without loss of generality, be its 

singular values and let X = X^^j^ o-iUivJ be the singular value decomposition of X. We define 

^max(r) = Sj=l '^i'^i^T ^^"^ max(r) = X — Xujax(r) = Si=r+1 ^i'^^i'^T • 

For < a < oo define the norm of a vector G as \\v\\a = (XliLi l"*^*!")"^^"- I'^ addition, 
||f||oo = supjlujl and \\v\\q = |supp(7;)|. For matrices X = (xij), Y = {yij) G M™^", define the 
inner product of X and Y as {X,Y) = trace(X-^y) = J2iLi'^]=iXijyij- The norm associated 
with this inner product is the Frobenius norm, ||X||i? = -s/ {X, X) = ^Yll^i X]j=i ^Ij- Note 
that M"*^" associated with this inner product is a Hilbert space. The spectral norm of a matrix 
X G '^rrixn jg defined as ||X|| = sup^g^n ||X/3||2/||/3||2, which is equal to the largest singular 
value of X. 

For a hnear map M : R™^" Ri, we denote its adjoint operator by M* : R'^ M"^^", so 
that for ah X G M™-x" and b G M^, {X,M*{b)) = {M{X), h)i^. For any given norm |-| in an inner 
product space (M"*^", (•, •)), the dual norm | • |d is defined as \X\ci = max{(X, Y) : \Y\ = 1}. It is 
well known that the dual norm of the Frobenius norm is itself and the nuclear norm and spectral 
norm are dual norms of each other. The null spaces of a matrix A G M"^*' and a linear map 
M ■ M'nxn ^ j^^g denoted respectively hy N{A) &ndN{M), i. e., M{A) = {/3 G : A/3 = 0} 
and M{M) = {X e M'"^" : M{X) = 0}. 

Finally, we introduce a useful tool for providing conditions for the exact recovery. Stojnic, 
Xu, Hassibi |19j gave a necessary and sufficient condition on the null space for the exact recovery 
of A;-sparse signals in the noiseless case. It was shown that one can recover all /c-sparse signals 
13 using dSD with B = {0} if and only if for ah p G N{A) \ {0}, 

ll/5max(fc)l|l < 11/5- max{fc) 111- (*) 

Oymak, Hassibi [TTj gave a similar result for the exact recovery of r-rank matrices in the noiseless 
case. One can recover all r-rank matrices X using Q with B = {0} if and only if for all 
X G AA(A^) \ {0}, 

ll^max(r) II* < 11^- max(r) II * • (**) 

Based on these results, one can consider the recovery problem by investigating the null spaces 
of A and M instead of checking the original definition of exact recovery, which often simplifies 
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the problem. 



3 Sharp RIP conditions for Sparse Signal and Low-rank Matrix 
Recovery 

With the preparations given in Section [21 we estabhsh in this section the main results of this 
paper ~ a sharp RIP bound for the exact recovery of sparse signals and low-rank matrices in 
the noiseless case and the stable recovery in the noisy case. A unified treatment is given for 
the sparse signal recovery and low-rank matrix recovery. We first introduce in Section 13.11 an 
elementary but important technical lemma which we call the Division Lemma, and then discuss 
the main results for sparse signal recovery in Section 13.21 and the low-rank matrix recovery in 
Section 13. 3i 

3.1 Division Lemma 

As discussed in Section [21 we will establish the RIP condition for the exact recovery using the 
null space properties of A and Ai. In order to relate the general elements in the null space with 
the RIP condition whose constraint is on the sparse vectors and low-rank matrices, a natural 
approach is to divide these elements into sums of sparse or low-rank components. Consequently, 
we introduce the Division Lemma below, which is a key technical tool for the proof of the main 
results. 

Lemma 3.1 (Division Lemma) Let r and m he positive integers with m >2r. Let ai > 02 > 

03 ^ • • • ^ flm > be a sequence of non-increasing real numbers satisfying 



r 



m 




(8) 



w=r+l 



Then there exist non-negative real numbers {sjj}i<i<r,2r+i<j<m such that 



r 




\f 2r -\- 1 < j < m, 



(9) 



i=l 



and 




V 1 < f < r. 



(10) 



w=l 



j=2r+l 
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The proof of Lemma [3. II is simply by induction on m. The Division Lemma can be illustrated 
as in the following table. Each row is an inequality; every element in the first row equals the 
sum of remaining elements in the same column: 



ai 


a2 


a,. 


> 




■ a2r + 


«2r+l 




ai/r 


a^jr •• 




> 
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•Sl,2r+1 




ai/r 


a2lr ■■ 


■ ar/r 


> 


ar+2 


+ 


•S2,2r+1 


S2,m 








> 




+ 






ai/r 


a2/r ■■ 


ar/r 


> 




a2r + 


Sr,2r+1 


Sr,m 



3.2 Sparse signal recovery 

We begin with the noiseless case {z = 0) of the sparse signal recovery model ([T]). In this case, 
The commonly used £i minimization method is by ([3]) with B = {0}. We shall present the sharp 
RIP condition on 6^ for the exact recovery of all A;-sparse signals for any given integer k >2. 

The following theorem shows that the condition 6'^ < 1/3 is sufficient for the exact recovery 
of fe-sparse signals in the noiseless case. 

Theorem 3.1 Suppose the measurement matrix A E MJ^^p satisfies < 1/3 for some integer 
2 < k < p. Let y = A(3 where 13 £ W is a k-sparse vector. Then the minimizer /3 o/ ([3]) with 
B = {0} recovers (3 exactly, i.e., (3 = (3. 

The result below shows that the condition 5^ < 1/3 is sharp for the exact recovery in the 
noiseless case. 

Theorem 3.2 Let 2 < k < p/2. There exists a measurement matrix A G M""^^ with = 1/3 
such that for some k-sparse signals ■j, rj £ with ^ rj, A-f = Ar]. Consequently, it is not 
possible for any method to exactly recover all k-sparse signals /? based on {A,y) with y = A(3. 
Ln particular, the £i minimization ^ with B = {0} cannot recover all k-sparse signals. 

Theorems 13.11 and 13 . 21 together show that the condition 5^ < 1/3 is sharp for all 2 < k < p/2. 
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Remark 3.1 In the above theorems, The case A; = 1 is excluded because the RIP cannot provide 
any sufficient condition for the exact recovery via the constrained ii minimization in this case. 
Take, for example, n = p - 1 > 1. Let ^ G M"^p with A/3 = (/3i - 02,^3:^4, • • • , for any 
/3 = (/3i , /32 , /Ss , • • • , /5p)"^ G K^- Then for all 1-sparse vectors /3, 

P/3||i = E/?'- 2/31/32 = ||/3||i, 

which implies the restricted isometry constant = 0. However, b = = Ar] where 7 = 
(1, 0, • • • , 0) and r] = (0, —1, 0, • • • ,0) are both 1-sparse signals. Thus it is impossible to recover 
both of them exactly relying only on the information of {A, b). In particular, the ii minimization 
([3]) with B = {0} cannot recover all 1-sparse signals. Since 6f = 0, the RIP cannot provide any 
sufficient condition in this case. 

We shall now turn to the noisy case of the sparse signal recovery model ([1]). The noiseless 
case provides much insight to the noisy case. In this case the error vector z is nonzero and we 
shall consider two bounded noise settings 

B'^iv) = {z:\\z\\2<v}, (11) 
e^^(r/) = {z:\\A^z\\^<7]}. (12) 

The case of Gaussian noise, which is a canonical model in statistics, can be treated similarly. 
See Remark 13.21 below. In the noisy case we shall also consider more general signals /3 which 
are not necessarily fc-sparse. Decompose /3 = /3niax(A;) + l^-max{k)- The norm minimization 
approach for recovering /3 in these bounded noise settings is by solving ([3]) with B = B''''^{ri) or 

B = B^^{-n). 

We first consider the stable recovery of /3 with the error z in a bounded £2 ball. 

Theorem 3.3 Consider the signal recovery model ([T]) with \\z\\2 < £■ Let (3 be the minimizer 
of §1 with B = B^^{r]) defined in for some 7] > e. If 6 = 6^ < 1/3 with k > 2, then 

1 — 3o i — 3o ^/k 

In particular, for all k-sparse signals (3, 
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The result is similar if the error z is in the bounded set H^'^zHoo < £• The i\ minimization 
method with B = is called the Dantzig Selector. See Candes and Tao [TJ. 

Theorem 3.4 Consider the signal recovery model ([T|) with z\\oo ^ £• Let j3 he the minimizer 
of © with B = B^^{r]) defined in ([ED for some r]>e. If S = 6^ < 1/3 with k>2, then 

- g|b < -^(e + ,) + ^^P^' + Vra?) + 2(1 - U) 11/3- ,.|||. ^ ^j^, 

1-30 1-30 ^/k 

Remark 3.2 Since Gaussian noise is essentially bounded, the results for the signal recovery in 
Theorems 13.31 and 13.41 can be directly applied to the Gaussian noise case. Interested readers are 
referred to Section 4 in [2] and Lemma 5.1 in [1] for details. 



3.3 Low-rank matrix recovery 

We now turn to the affine rank minimization problem. As mentioned before, the results are 
parallel to those for the sparse signal recovery. As in Section 13.21 we begin with the noiseless 
case. The ideas and results can be extended to the noisy case later. Consider the matrix recovery 
model ([2]) with z = 0. The nuclear norm minimization method in this case is given by (jH) with 
B = {0}. The goal is to recover the matrix X whose rank is at most r. 

For the same reason as in the signal recovery problem, we shall only consider the case r > 2. 
The following two theorems, which are parallel to Theorems 13.11 and 13.21 are the main results in 
this paper for the low-rank matrix recovery. Theorem 13.51 shows that the condition < 1/3 is 
sufficient for the exact recovery of r-rank matrices. 

Theorem 3.5 Suppose 2 < r < min(m,n). Let X be a matrix of rank at most r and let 
b = ^A{X). If < 1/3, then the solution X^ of the nuclear norm minimization (jl]) with 
B = {0} recovers X exactly, i.e., X^ = X. 

The following theorem shows that the condition 6{!^ < 1/3 is sharp. These results together 
establish the optimal bound on for the exact recovery in the noiseless case. 

Theorem 3.6 Let 2 <r < min(m,ri)/2. there exists a linear map A4 with 6^ = 1/3 such that 
for some matrices X,Y ^ j^mxn ^^^^ rank(X), rank(y) < r, ^A{X) = M(Y). Consequently, 
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there does not exist any method that can exactly recover all matrices of rank at most r based on 
{A4,b) with b = j\4{X). In particular, the nuclear norm minimization ^ with B = {0} cannot 
recover all r-rank matrices. 

We should note that the result above is stronger than Theorem 1.2 in Wang and Li [20] as 
it shows that there exists some linear map A4 with 5:^ = 1/3 such that all methods, not just 
nuclear norm minimization, fail to recover all rank r matrices in the noiseless case. 

Remark 3.3 The reason for excluding the case r = 1 in the two theorems given above is the 
same as that in the signal recovery problem: the RIP cannot provide any sufficient condition in 
this case for the exact recovery through the nuclear norm minimization. An example is given as 
follows. Let m,n>2 and let the linear map A4 : M™^" — ). ]^rnn-2 defined by 

M{X) = {Xii - X22, Xi2 + 2:21, Xis, • • • , Xin, X23 ' ' ' , X2n,X3l, ■■■ , Xmnf 

for X = (xij) e R"*^". Then for all matrices X such that rank(X) < 1, 

m n 

\\M{X)\\l = J2^xl - 2(xiiX22 - rEi2X2i) = \\X\\l. 

1=1 j=l 

This implies the restricted isometry constant (5^ = 0. In addition, one can check that X = 
diag(l,0,--- ,0), y = diag(0, -1,0,--- , 0) are both of rank 1. In addition, 6 = = M{Y). 

This means that the exact recovery is impossible based on {M,b) in the noiseless case. Hence 
the RIP cannot provide a sufficient condition to ensure the exact recovery of all matrices with 
rank at most 1. 

We now turn to the noisy case. As in the signal recovery problem, we also consider bounded 
noise in two settings 

B^^{r]) = {z : \\z\\2 < r?}, (15) 
B^^{7]) = {z : \\M*z\\ < r]}. (16) 

We shall also consider general matrices that are not necessarily exactly low-rank. Decompose 
X = -'^^max(r) + -^-max(r)- The nuclear norm minimization method is to recover X by solving 
(HD with B = B^^ri) or B = B^^{ri). 

We first consider the case where the error z is in a bounded £2 ball, ||2:||2 < e. 
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Theorem 3.7 Consider the affine rank minimization problem ([2]) with \\z\\2 < S- Let X^: be the 
minimizer of (jH) with B = B^'^ (77) defined in (jlSp for some rj > e. If 5^ < 1/3 with r>2, then 

\\Y Y\\ ^\/2(rT5) , , 2V2(2<^+^(l-3J)<^) + 2(l-3J)||X,n.ax(r)ll* 

11^* -X\\f< i_3^ (e + + ^ . (17) 

For matrix recovery under the model ([2]) with the error bound ||A^*(z)|| < e, we have the 
foUowing similar result for the matrix Dantzig Selector. 

Theorem 3.8 Consider the affine rank minimization problem ([2|) with \\Ai*{z)\\ < e. Let X^, 
be the minimizer of (HD with B = B^^{rj) defined in (|16p for some rj > e. If 5^ < 1/3 with 
r > 2, then 

\\Y Y\\ ^ ^ , ^ , 2^/2(2,5 + v^(l-3JM) + 2(1- 3,5) II, 

11^* -Xy< ^-^(6 + r?) + . (18) 

We omit the proof of Theorem I3.8|. which is essentially the same as that of Theorem 13. 7i 

Remark 3.4 Similarly as in the sparse signal recovery problem, the results for the low-rank 
matrix recovery in Theorems 13.71 and 13.81 can be extended to the Gaussian noise case. The 
readers are referred to Lemma 1.1 in Candes and Plan [9] for details. 



4 Oracle inequalities and RIP conditions on (5^^ and 

Oracle inequality provides great insight into the performance of a procedure as compared to that 
of an ideal estimator. It was first introduced in Donoho and Johnstone [H] in the context of 
statistical signal processing using wavelet thresholding. This method has since been applied in 
many other problems. In particular, various oracle inequalities have been given in the literature 
for the constrained ^i/nuclear norm minimization procedures. See, for example, Candes and 
Tao [7], Cai, Wang and Xu j3], and Candes and Plan [9]. Theorem 14.11 below provides oracle 
inequalities for sparse signal and low-rank matrix recovery under the condition 5^ < 1/3 and 
(5^ < 1/3 given in this paper. The technique is analogous to the one used in Candes and Plan 
[9], along with Lemma l4. 1 1 given below. Theorem 13.41 and Theorem 13.81 
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Theorem 4.1 Given the signal recovery model ([T]), suppose z ~ Np{0,a'^I) and the signal 
/3 S is k-sparse. Assume that /3 is the minimizer of ([3]) with B = {z : \\A^ z\\oo < A = 
4o-V(2/3)logp}. If 5^ < 1/3 with k > 2, then 

11/3 - /3i < _ 3^^^^ logp ^ mm(/32, (19) 

with probability at least 1 , ] 

Similarly, for the matrix case ([2|), suppose z ~ Nq{0,a'^I) and rank{X) < r. Assume that 
is the minimizer of O with B = {z : \\M*{z)\\ < A = 16a^(l/3) log(12) max(m, n)}. // 
6{J^ <l/3 with r>2, then 

11^* -^IIf < ^^^^^^^ j;min(a2(X),max(m,n)a2) (20) 

lyii/i probability at least 1 — e~'^'^^^(™''"), where c > is an absolute constant, and ai{X), i = 
1 • • • , min(m, n) are the singular values of X. 

We should note that the main ideas for the proof here are essentially the same as those for 
the proof of Theorem 2.6 in [9], where readers can find more details. Finally, it is noteworthy 
from these oracle inequalities that in the case of /3 = or X = 0, i.e., the input signal or 
matrix is identically zero, the Dantzig Selector recovers the zero input exactly by zero with high 
probability in the Gaussian noise case. 

In addition to providing the sharp condition on 5^ and 5^, the techniques developed in 
this paper can also be applied to sharpen other RIP conditions such as 5^ and (5^ for the 
exact/stable recovery of the sparse signals and low-rank matrices. Since < 1 is known as a 
necessary condition for the model identifiability (see Lemma 1.2 in [6]), much previous attention 
has been on the bounds for and 5^ as the sufficient conditions for the recovery of the sparse 
signals and low-rank matrices. Applying the same method as that used in the previous section 
on 5^ and 5^, we have the following theorem for (5^ and 5^^ . 

Theorem 4.2 Suppose 1 < k < p. Let y = A/S for a k-sparse vector /3 G RP. If S^j^ < 1/2, then 
the minimizer (3 of ^ with B = {0} recovers /3 exactly, i.e., (3 = 13. 

Similarly, suppose 1 < r < min(m,n) and let b = A4{X) for some matrix X with r-rank. If 
5^ < 1/2, then the minimizer X* of 1^ with B = {0} recovers X exactly, i.e., X* = X. 
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To the best of our knowledge, these are the best bounds on 5^ and (5^ available as a sufficient 
condition for the exact recovery of the sparse signals and low-rank matrices, respectively. Note 
that Davies and Gribonval [10] proved that it is not possible to exactly recover all A;-sparse 
signals in the noiseless case when (5^ > \/2/2. Hence, the upper bounds on are necessarily 
less than y/2/2. There is still a gap between the two bounds 1/2 and \/2/2 on 5^^. It is an 
interesting future project to close this gap. 

It is also noteworthy that Zhang ([21], Remark 1) proved for some concave penalty the 
estimator 

/3 = argmm (^\y - Xp\\l + ^p(|/3,|, A)^ 

recovers /c-sparse signals exactly in the noiseless case with a suitable choice of A under the 
condition 52k < 1/2 or 63k < 2/3. The constrained £1 minimization estimator f3 defined in ([3|) 
with B = {0} is straightforward to compute. In contrast, the concave penalized minimization 
estimator requires a good choice of the tuning parameter A and is not as easy to implement. 

It is also interesting to consider conditions on 5^ and for some integer s > 1. The 
following result provides convenient bounds on and (5^ in terms of (5^ and 5^ respectively. 
It is also useful for the proof of Theorem 14.11 

Lemma 4.1 For all matrix A G M"^^ and k > 2, s > 2, we have (5^ < (2s — 1)5^. Similarly, 
for all linear map M : M"''" -^W andr>2, s> 2, we have 6^ < {2s - 1)6{^ . 

5 Proofs 

In this section we shall first prove the main results. The proofs of some of the main theorems 
rely on a few additional technical lemmas. These technical results are collected and proved in 
Section 15.91 

5.1 Proof of Theorem 1331 

The key to the proof of this theorem is parallelogram identity, since it provides equality rather 
than inequality in the estimation in ^2 norm as we shall see later. 
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By (**), we only need to show for all R £ AA(7W)\{0}, it satisfies ||i2max(r) II* < ll-R- max(r) II*- 

Suppose there exists R £ J\f{M) \ {0} such that ||i?max(r) II* ^ ll-R- max(r) II*- Assume R has 
SVD decomposition R = Y^^i o-i'^J'^i^ cli > a2 > ■ ■ ■ > a^- Since we can set Oj = if i > m, n, 
without loss of generality we can assume that m,n > r. 

By Lemma IXTl we can find {sij}i<j<r,2r+i<j<m satisfying ([9]) and (fTOl) . 
1. When r is even, suppose 

r/2 r 3r/2 2r 

Ru = y^^ajUivf, Ri2= ^ aiUivf, R21 = ^ atUivf , R22 = ^ arnvf 

i=l i=r/2+l i=r+l i=3r/2+l 

m r/2 m r 

R31= Yl C^^ijUj^J)^ ^32= X] ( SijUjvJ) 
j=2r+l i=l j=2r+l i=r/2+l 

(21) 

then M{Ru + Ru + R21 + R22 + R31 + R32) = M{R) = 0. By the parallelogram identity, 

\\M{-Ru + R22 + i?32)f + \\M{-Ri2 + R21 + R3l)f 
=^ [WMi-Rii - R12 + R21 + R22 + R31 + R32)f + \\M{-Rii + R12 - R21 + ^22 -R31+R 



\\M{2Ru + 2Ri2)f + ^\\M{-2Rn - 2R21 - 2i?3i)f + \\\M{2Ri2 + 2ii22 + 2R32) 



=2\\M{Rii + Ri2)f + \\M{Rii + R21 + R3i)f + \\M{Ri2 + i?22 + R32)f 

We use Lemma 15.21 by setting 

g = h = r/2, bi = ai,Ci = -ai^r/2,di = ai+r, VI < i < r/2, 

r 

Cj = ^ Sij+2r, Uj = Sij+2r, 1 < « < r/2, 1 < j <m- 2r, 

i=l 

then we get 

\\M{Ru + R21 + R3i)f - \\M{-Ri2 + R21 + R3i)f 

r/2 3r/2 m r 3^/2 

i=l i=r+l j=2r+l i=r/2+l i=r+l j=2r+l 

(23) 
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(22) 



m. 



Similarly, 

\\M{Rl2 + R22 + R32)f - \\M{-Ru + R22 + R32)f 

r 2r m r/2 2r m 

j=r/2+l i=3r/2+l j=2r+l i=l i=3r/2+l j=2r+l 

(24) 

Let the right hand side of (j22|) minus the left hand side. Along with (|23|) . (|24|) . we get 
= RHS-LHS 

r r 2r m 

i=l i=l i=r+l j=2r+l 



> 2(l-25,^)^a2-25,^r I'&i^y 

1=1 ^ ^ 

r 

> 2(1-35,^) a? 

i=l 



The last two inequalities are due to (jlOp and Cauchy-Schwarz inequality. It contradicts 
the fact that R and 5^ < 1/3. 



2. When r is odd, r > 3, note 



(r+l)/2 



-Rii 


= aiuivj, R12 


= E 




Rl3 


= E 


aiUivf 








i=2 






j=(r+3)/2 












{3r+l)/2 






2r 




-R21 


T 


R22 = 


E 




^23 = 












j=r+2 




i= 


=(3r+3)/2 






m 




m 


{r+l)/2 




TO 


2r 


-R31 




-R32 = 


= E 


( E 


\ T 
SijJUjVj , 


^33= E ( 






j=2r+l 




j=2r+l 


1=2 




j=2r+l j= 


= (r+3)/2 



(25) 

Note Xi = -Ru + i?2i + -R31, ^2 = --R12 + ^22 + ^23, ^3 = -^13 + ^23 + ^33, we Can 
easily show the following equality 

A\\M{X,)f + A\\M{X2)f + 4\\MiX,)\\' 
= \\M{Xi+X2 - X3)f + \\M{-Xi + X2 + Xs)f (26) 
+ \\M{Xi -X2 + X^)f + \\M{Xi +X2 + X^)f 
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By the fact that M{R) = 0, ([26]) means 

\\M{-Rn + R21 + i?3i)f + \\M{-Ri2 + R22 + R32)f + \\M{-Ri3 + R23 + Rss)^ 
= \\M{Ri2 + Ri3 + R21 + i?3i)f + \\M{Rn + R13 + R22 + R32)f 
+ \\M{Rn + R12 + R23 + R33)f + \\M{Rii + R12 + Ri3)f 

(27) 

Similarly as the even case, by Lemma 15.21 we have 

\\MiR12 + Ri3 + R21 + R3i)f - \\M{-Ru + R21 + R3i)f 



>(l-^i 



j=2r+l 



i=2 



(1 + 5.^) 



al + (a,.+i + ^ sij)^ 

j=2r+l 



(28) 



||A^(i?ll + Rrs + i?22 + i?32)f - \\Mi-R12 + i?22 + R32)\\' 



(r+l)/2 



>(l-<5;^) 
-(1 + ^.^) 



i=(r+3)/2 4=2 j=2r+l 

■(r+l)/2 (r+l)/2 m 



(29) 



i=2 



i=2 j=2r+l 

2 II A/(/ D , D , D Ml2 



||A^(i?ll + i?i2 + i?23 + ^33)lr - \\M{-Ri^ + i?23 + R33)\\ 
>(l-5.^) 



(r+l)/2 r m 

E E («^+ E 



i=(r+3)/2 j=2r+l 



(30) 



(1 + 5.^) 



E E («^+ E 



_t=(r+3)/2 i=(r+3)/2 j=2r+l 

Let the right hand side of (|27|) minus the left hand side, we can get 



> {1-5^) 



^E"^*^ + E(°^+^ + E 

j=2r+l 



j=l 1=1 



(1 + 5.^) 



E"? + E("^+*+ E 

j=2r+l 



j=2r+l 
2 



i=l i=l J 

> 2(l-25,^)^a2-2<^,^r r&i^') 

> 2(l-35,^)^a^ 



i=l 
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The last two inequalities are due to (|10p and Cauchy Schwarz inequality. It contradicts 
the fact that / and (5^ < 1/3. □ 



5.2 Proof of Theorem EH] 

The proof of Theorem 13. II is essentially the same as Theorem 13.51 By (*), we only need to show 
for ah /3 G J\f{A) \ {0}, it satisfies ||/3max(fe) 111 < 11/?- max{fc) 111- 

For the convenience of presentation, we call a vector with 1 or -1 in only one entry and zeros 
elsewhere as the indicator vector. 

Suppose there exists /3 G A^(^) \ {0} such that ||/3max(/c)lli < 11/^- max(/c) 111- Then j3 can be 
written as 

p 

/5 = E' 



p 

am 
1=1 

where {ui}^^^ are indicator vectors with different support in W; {ai}^^^^ is a non-negative and 



decreasing sequence. Since we can set ai = if i > p, without loss of generality we can assume 
that p > k. 

By Lemma |3. 11 we can find {sij}i<i<k,2k+i<j<p satisfying ([9]) and (jlOp with a modification 
of notations. 



1. When k is even, suppose 

k/2 k 3k/2 2k 

/3ii = '^aiUi, /3i2= ^ OiUi, (321= ^ am, (322 = ^ am 

i=l i=k/2+l i=k+l i=3fc/2+l 

p k/2 p k 

nU 



(31) 



j=2fc+l i=l j=2k+l i=k/2+l 

then A{(3ii + (3i2 + (32i + (322 + /^si + (332) = A(3 = 0. By the parallelogram identity, 

\\A{-(3u + (322 + (332)f + \\A{-(3i2 + (321 + /33^"' 



^31) 

[P(-/3n - (3i2 + hi + /322 + 1331 + /332)f + P(-/3ii + /3l2 - /321 + /322 - 1331 + /332)l"^ 
P(2/3n + 2/3i2)f + ^P(-2/3n - 2^^ - 2(3^i)f + \\\A{2h2 + 2/322 + 2/332) 



=2P(/3n + /3i2)f + P(/3ii + hi + hi)\? + \\A{Pi2 + P22 + ^32)^ 

(32) 
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Similarly as the matrix case, we use Lemma 15.21 and get 

Wii + hi + ki)f - \\A{-(3i2 + hi + hi)f 

k/2 3k/2 p k 3k/2 p 

i=l i=k+l j=2k+l i=k/2+l i=k+l j=2k+l 

(33) 

Similarly, 

\Wl2 + h2 + /332)f - \\A{-hl + h2 + /332)f 

>{i-st){ E E («^+ E s,,f)-{i + 6t){Y,a^+ E K+ E ^ 

i=k/2+l i=3fc/2+l j=2fc+l i=l i=3fc/2+l j=2k+l 

(34) 

Let the right hand side of ([32]) minus the left hand side. Along with ([33|) . we get 
= RHS-LHS 

k k 2k p 

> 2(l-4^)(Ea?)-24^E"?-24^( E («^+ E ^*^)') 

i=l i=l i=k+l j=2k+l 

> 2(l-2<^,^)E«?-2<5^f%^') 



> 2(1- 35,^) E 



i=l 
k 



i=l 

The last two inequalities are due to (jlOp and Cauchy-Schwarz inequality. It contradicts 
the fact that /3 / and 5^ < 1/3. 

2. When k is odd, /c > 3, note 

(fc+l)/2 fc 

/3ii=ai'Ui, /3i2 = E -^13= E 

j=2 i=(fc+3)/2 
(3fc+l)/2 2k 

P21 = ak+iUk+i-, P22 = E ^23 = E "^^^^ (^^) 

i=k+2 i=(3fc+3)/2 
p p (fe+l)/2 p 2k 

p3i = E ^ij^j' ^32 = E ( E /333 = E ( E 

i=2fc+l j=2fc+l j=2 j=2fc+l i=(fc+3)/2 
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P(/3i2 + /3i3 + hi + Pzi)f - + hi + /3; 



'31 J 



>(1 - 



j=2k+l 



1=2 



(1 + 



j=2k+l 



(36) 



Note 71 = -j3n + /32i + /Ssi, 72 = -/3i2 + /322 + /323 , 73 = -/5i3 + hs + /333, we can easily 
show the fohowing equahty 

4P7i||2 + 4P72|p + 4P73f 

= P(7i + 72 - 73)f + P(-7i + 72 + 73)f 
+ P(7i - 72 + 73) f + P(7i + 72 + 73) f 
By the fact that A/3 = 0, (I36p means 

P(-/5ll + /321 + /33l)|P + P(-/3l2 + /322 + /332)f + P(-/5l3 + /323 + hs)^ 
= p(/3i2 + /3i3 + P21 + /33l)f + + /3l3 + /322 + h2)f 

+ P(/3n + /3i2 + /323 + /333)f + + /3i2 + Mf 

Similarly as the even case, by Lemma 15.21 we have 



(37) 



(38) 



P(/3ll + /3l3 + /322 + /332)|r - P(-/3l2 + (322 + M 



>(l-4^) 



-(1+5.^) 



(fc+l)/2 

i=(fe+3)/2 i=2 j=2fc+l 

■(fc+l)/2 (fc+l)/2 p 



i=2 i=2 j=2k+l 

2 W Al a 1/3 I « M|2 



P(/3ii + /3i2 + /323 + kiW - P(-/3i3 + /323 + 

k p 



(fc+l)/2 

E E («^+ E 



Si 



i=l i=(fe+3)/2 
k k 



j=2k+l 



E E E •'^^j')' 



j={fc+3)/2 j={A:+3)/2 j=2k+l 

Let the right hand side of p7p minus the left hand side, we can get 

k 



> (1-5,^) 



k k p 

3 E + E + E 

i=l i=l j=2k+l 

k k p 

i=l j=2fc+l 



(1 + 



(39) 



(40) 



E"*^ + E("'^+*+ E 

j=2fc+l 



i=l 1=1 



i=l 
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i=l \ 

k 



i=l 

The last two inequalities are due to (jlOp and Cauchy Schwarz inequality. It contradicts 
the fact that /3 / and 5^ < 1 /3. □ 

5.3 Proof of Theorem ISTHl 

It is well known that for matrices X, B with the same size, B)\ < \\X\\f\\B\\f. The following 
lemma provides a stronger result given further constraint on matrix rank. 

Lemma 5.1 Let X E R'"^"(m < n) be a matrix with singular values Ai > A2 > • • • > Xm, then 
for all B G ]R™x" such that rank{B) < r, we have 



{B,X)| < ||B||p. 



\ i=l 



Proof of Lemma 15.11 Since the rank of B is at most r, we can suppose B,X have singular 
value decomposition B = UT.V ,X = WkZ, where U,W ^ R'"><™,S,A G M™><",y,Z G M"^". 
Then 

{B,X) = tr{B^X) = tr{V'^T,^U'^WAZ) = tr{T,^U^WAZV'^) = diag(S) • diag{U'^WAZV^) 
Since the rank of B is at most r, diag(Ti) is supported on the first r entries, 

\{B.X)\< 



Y,{U^WKZVT)l < \\B\\p^ 



Y^J2{UTWAZVT)l = \\B\\f\\KAZV^\\f 



\ i=l \ 1=1 

where we note K G M''^" as the first r rows of U'^W. In addition, 

WKAZV'^Wl = tr{VZ'^A^K'^KAZV'^) = triAZV'^V Z'^ A^ K'^ K) = tr{A^K'^K) 

By K is the first r row of an n x n orthogonal matrix, we have tr{K'^ K) = tr[KK'^) = tr{Iy) = r 
and all diagonal elements of K^K are in [0, 1], then 

n r 

tr{A^K^K) = Y,Xj{K^K),.<Y,Xl 

i=l i=l 
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In summary, 



\{B,X)\ < \\B\\f\\KAZV'^\\f < \\B\\f, 



\ i=i 



It is noteworthy that the signal version of this lemma simply holds by Cauchy-Schwarz 
inequality. 

Now we construct an example for Theorem 13.61 then check the feasibility by the lemma 
above. Note 

2r 



Xi=diag(-^,...,-^,0,---,0)GM™x" 

Suppose H = (M™^", IIXIIj?) is the Hilbert with inner product (•,•). Since HXiHi? = 1, we can 
extend Xi into a basis {Xi, • • • , Xmn}- Define M : M™''" ^ M"^" as 

/"T mn 

for all X = Y17=i ^i^i- 

Then by Lemma IS.lj for all matrix X with rank at most r, we have 



\{X,X^)\<-^r-^jX\\F = ^\\\X\\F 

A A A 

\\M{X)\\l = 3 = gdl^llF -«?) = 3(11^11^ - \{X,X,)\') 

4=2 

Thus, 

^||X||^<||A^(X)||2<^II^II^, S^{X)<l/3 

Notice that 

r r 

||A1(diag(l~^,0, • • • ,0))||2 = ^r = ^||(diag(l~^, 0, • • • ,0))||^ 

||M(diag(l, -1, 0, • • • , 0))||i = ^ = ^||diag(l, -1, 0, • • • , 0)111 
we can conclude that = 1/3. Finally, suppose 



X = diag(l, 1 • • • ,1,0,-- - ,0), y = diag(0,--- ,0,-l,-l-- - ,-1,0,- •• ,0) 

Then X, Y are both matrices of rank r such that X - Y e M{M), M[X) = MiY). Therefore, 
it is impossible to recover both X and Y only given (6, A^), which finishes the proof. □ 
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5.4 Proof of Theorems 13.21 



Again, the proof to this theorem is essentially the same as Theorem 13. 6i Note 

2k 



Suppose H = (MP, II • II2) is the Hilbert with inner product (•, •). Since ||/5i||2 = 1, we can extend 
Pi into a basis {/3i, • • • , j3p}. Define ^ : as 

1=2 

for all /3 = Yji=i <^if^i- 

Then by Cauchy-Schwarz inequality, for all /c-sparse vector 7, we have 

|(7,/3l>| < IIA • lsupp(7)l|2||7l|2 < y^hlb 

PTili = ^E«? = Uh\\2 - «?) = ^(hlli - K7,/3i)P) 

i=2 



Thus, 



Notice that 



^||7lli<P7lli<^ll7lli, S^<l/3 



\\ACi,--- ,1,0,--- ,0)||i = ^A: = ^||(1,--- ,1,0,--- ,0)||i 



P(l,-1,0,--- ,0)||2 = - = -||(l,-l,0,--- ,0)||2 



4 

3 3 

we can conclude that = 1/3. Finally, suppose 

k k 



7= - ,1,0,-- - ,0), r? = (O,--- ,0,-l,-l--- ,-l,0,-- - ,0) 

Then 7, 77 are both matrices of rank k such that 7 — ?y G M{A), A7 = Ai]. Therefore, it is 
impossible to recover both 7 and r] only given {y,A), which finishes the proof. □ 

5.5 Proof of Theorems Q and [37fl 

For the proof of Theorem 13 . 3 1 and Theorem [321 we only show the the latter one about the matrix 
case, as the proof to the signal case is similar and simpler. Suppose R = — X, h = (3 — (3. 
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We will use a widely used fact. The readers may see [T], [7], [8], [13] (signal case) or [20] (matrix 
case) for details: 

11^— max(fc)l|l — ll^max{A;) II* + max(A;) 111 

11^— max(r) II* — ll-^max(r) II ~l~ ^11^— max(r) II* 

For the remaining part of proof, we only prove the matrix case. Suppose R has singular value 
decomposition R = X]™ ^ QiUivJ . Then we have 



m 



+ 2||X_^a^(^)||^, > ^ ai (43) 

1=1 j=r+l 

Apply Division Lemma [3T] bv setting a[ = ai + 2||X_max(r) ll*/?^) ^ = 1) ' ' ' and a'^ = aj,j > 
r + 1, we can find {sjj}i<j<r,2r+i<j<m satisfying 

r 

^ Sij = ttj, \f 2r + 1 < j < m, (44) 

i=l 

I r Oil Y II 

- 2^ ai« H > a^+j + ^ij^ y I < I < r. (45) 

We also know 



r ^ — ' r 

ui=l j=2r+l 



||7W(i?)|| < ||A^(X) - 6|| + \\b -MiX,)\\ <e + r]. (46) 
Similarly as Theorem 13.51 we finish the remaining part of proof for even or odd r separately. 

1. When r is even, we define Rn,--- ,i?32 as ([2T]) . similarly as (p2|) and by parallelogram 
equality, we get 

\\M{-Ru + i?22 + ^32)11' + ||A^(-^12 + i?21 + R3l)f 
= \ [Il-M(-^11 - Rl2 + R2I + R22 + R3I + i?32)f 
+ ||A^(-i?ll + R12 - R2I + R22 - R3I + R32)f] 

=^||A^(2i?n + 2R12) -M{R)f + ^\\M{-2Rn - 2i?2i - 2i?3i)||2 (47) 

+ ^\\Mi2Ru + 2R22 + 2i?32)f - ^\\M{2R)f 
=2\\M{Rii + Ri2)f + \\M{Rii + R21 + R3i)f 

+ ||A^(i?i2 + R22 + R32)f - 2{M{R),M{Rii + R12)) 
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Let the right hand side of (jl7|) minus the left hand side. Along with (j23|) . ([Mj) . one get 
= RHS - LHS 



2r 



>2(l-5,^) J^a2-25,^ J^a2-2,5,^( (a,+ J] s,,)^) - 2{M{R), M{Rn + R12)) 

i=r+l j=2r+l 



i=l 



> 2(1 - 25,^) j2 - 2<5^;^r(&i^ + ^11-^— (-)ll* )2 _ + ^) 



1=1 



>2(l-25,^)j;af-2<5^;^( 



i=l 



i=l 



i=l 



(48) 



By (|18]) we can get an inequality of \ Yl^i=i ^1' 



i=l 



1-35 



+ 



(^ 2||X_^.(.)|U ^ £+2^/1^)2 + (1 _ 3J)J||2X_„,,(,) ||2/r (49) 

1 - 3(5 



< 



VT+6{e + r,) + 2(25 + ^(1 - 35)5)||X„^,,(,) 



1-35 



Finally, by Lemma 15.3^ 



=r+l 



2||X_ 



max(r) II* \2 



Then 





m 




II^IIf = . 










^=1 ^ 



2||X 

~max{r) II* n 9 ^ 



2E«? + 



2||X 

— max(r 



^ ^2(1 + 5) 2^2(25 + V(l- 35)5) + 2(1 -3(5) ||X_^aa;(,) II, 

- •- v*^ ~r ^/j 



1-35 



1-35 



(50) 



2. When r is odd, we use the definitions in ()25p . Similar equality as ()47p holds as follows, 
||A^(-i?ii + R21 + R3i)f + \\M{-Ri2 + R22 + R32)f + \\M{-Ri3 + R23 + Rss)^ 

= \\M{Ri2 + Rl3 + i?21 + i?3l)f + \\M{Ru + iil3 + ^22 + ^32)f 

+ \\M{Ru + R12 + R23 + R33)f + \\M{Rii + i2i2 + Ri3)f 
-2{M{Ru+Ri2 + Ri3,M{R)) 
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By the method in the even case, we can still get the inequality (j48p . Hence we have the 
same estimation. □ 



5.6 Proof of Theorem 14.21 

By (*), we only need to show for all R £ M{M.)\{0}, it satisfies ||-Rmax(r) II* < ll-R-maa;(r) II*- 

Suppose there exists R £ M{A4)\{0} such that ||-Rniax(r) II* ^ ll-R-max(r-) II*- Suppose R has 
singular value decomposition: o-iUivJ . Note: 

r 2r 3r m 

Ri = '^aiUivf, R2= aiUivf, ^3 = ^ aiUivf , i?c = ^ aiUivf (51) 

1=1 i=r+l i=2r+l i=3r+l 

Notice that Yll=i^i — Yl^r+i'^i — Si^2r+i'^«- addition, two equalities cannot hold simul- 
taneously since / 0. Thus, 

r m 
i=l i=2r+l 

Apply Lemma [XT] to {ai, - - - , a^, a2r-+i5 • • • , dm}, we can find {sjj}i<j<r,3r-+i<j<m such that 

r y^r ^ rn 

Sij = aj, V3r + I < j < m; — — — > a2r+i + Sij, Ml < i < r 

1=1 ^ j=3r+l 

By Ya=i > YT=2r+i th^'^^ exists I <i <r such that > a2r+i + Yl%3r+i ^ij- We 

also have the equality in I2 space as follows, 

6||A^(i?i + R2)f + 3\\M{Ri + R3 + Rc)f = 2\\M{-R2 + R3 + Rc)f + \\M{3Ri + 2R2 + R3 + Rc 

= 2\\M{-R2 + R3 + Rc)f + \\Mi-Ri +R3 + Rc)f 

(52) 

Let the left hand side of (j52p minus the right hand side, by Lemma 15.21 we get 

= Q\\M{Ri+R2)f + 2{\\M{Ri+R3 + Rc)f -\\M{-R2 + R3 + Rc)f) 
+ (||A^(iii + R3 + Rc)f - \\M{-Ri + R3 + Rc)f) 



2r 



> 6(1-05] af + (l-0 2j]a2 + 3j](a2.+.+ ^^.)' + E 

i=l \ i=l i=l j=3r+l i=l 



2r r m r 

2 



-(1 + 5^) |2 J] a2 + 3j](a2.+.+ ^^.O' + E"^ 

i=r+l 1=1 j=3r+l i=l 
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r 2r r m 

i=l r+1 i=l j=3r+l 



i=l j=l j=3r+l 



2 



> 3(e«?-(^)^) 



> 



which is a contradiction. □ 
5.7 Proof of Lemma 14.11 

We only show the matrix case. For all X G ]^™x" such that rank{X) < 2r, suppose X has 
singular value decomposition X = Yli=i ^i'^i'^I ^ ^ — Without loss of generality we can 
assume / = sr as we can set Oj = if / < i < sr. Note 

Wi = M{aiUivf) eW, 1 <i< sr 

We can verify the following identity 

sr ^ 
II \ ^ ii2 \ ^ II ||2 

Z^W^2 + > Wi-Wj2 

^-^ sr — 1 ^-^ 

i=l l<i<j<sr 

sr ^ 
= (l + (s-l))^||^,;,||2 + 2(l-^^) (^''^J') 
i=l l<i<j<sr 



S 



2 



E ll'^il +^12-1 \-W. 



|2 
ir Il2 



l<ii<---<ir<:Sr 



which implies 

sr 

\\Mix)\\l = iiE^^lli 



i=l 



^rJ l<^^<■■■<ir■<sr 



l<ii<---<ir<:Sr l<i<j<sr 
rs 

(s(i + 5,^)-(s-i)(i-5,^))E«' 

i=l 

(1 + (2,-1)5^^)11X11^ 



\\MiX)\\l = 



i=l 
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^»'/ l<n<---<V<sr l<i<j<sr 

rs 

i=l 

(I - (2s - l)S^)\\X\\l 



Hence, 5^ < (2s - □ 
5.8 Proof of Theorem 14.11 

By a small extension on Lemma 5.1 in [1], we know ||^-^z||oo < a^J{l + 5^)logp < A/2 with 
probability at least l/-v/7rlogp. While for the matrix case, by Lemma 1.1 in [9], we know 
\\M*\\ < Aa^Jmax{m, n)(l + 5^^) log 12 < A/2 with probability at least 1 - e'='^'^^(™'"). Then in 
order to finish the proof, we only need to show p9|) or (j20p given the assumption ||A^2;||oo < A/2 
or ||7W*(z)|| < A/2. For the following part, we only give the proof for the signal case, since the 
matrix case is similar and the original proof by Candes and Plan in [9| is already for the matrix 
case. Define 

Kit P) = lUh + W - ml l = — = 2a^ \ogp 

lb 

Let ^ = argmin5K(^,/3), then we can deduce \\P\\q < ||/3||o < A; by K(^,/3) < K{p,P). By 
Lemma 14. H 



By Lemma 15.41 we have 

\\A^{y - A^)\\^ < \\A^{y - yl/3)|U + - /3)||oo < A 

which implies we can apply Theorem 13.41 by plugging /3 by /3: 



< V«.2A 



Hence, 



27 



Suppose /3' = X^iLi ftl{|/3i|>At}ej, where ei is the vector with 1 in the iih. entry and elsewhere, 
KCP.P) < i^(/3',/3)<7X^l{|ft|>;.} + P/3-^/3'||i 

i=l 

P P P 

< 7 E i{ifti>/^j + (1 + ^k) E i{ifti<M}iAi' ^ E (1 + -^fc 

i=l 1=1 i=l 

i=l 

The last inequality is due to 21ogp > {1 + 6^). In summary, we get (|19p given the assumption 
1 1 ^"^-2 1 loo ^ A/2, which finishes the proof. □ 



5.9 Technical Lemmas 

As seen in the proofs of Theorems 13.11 and 13.51 it is necessary to estimate the left hand side of 
(I23D, (IMD, (IMD, (I22D and dSUD- Notice that these terms are of the similar type - they are all 
the differences of the squared Frobenius norm of two matrices which only differ on a few leading 
terms in their SVD decompositions, we have the following lemma for the general estimation of 
this type of differences. Before we present the lemma, recall that we have defined the concept 
of indictor vector in Theorem 13.11 



Lemma 5.2 For the vector case, suppose g,h> 0,g + h < k, {di}f^^, {cj}^-^]^, {tjj}i<j<g^i<j<i 
are non-negative real numbers satisfying 

min di > max Cj, (54) 
1<*<9 i<i<i 

9 

Y^tij = ej, yi<j<l (55) 
1=1 

{hi}^^i^ {ci}^^i are real numbers, {uu,--- , ttih; ttsi, • • • ^u^g^un, - ■ ■ ,1*4;} is a set of indicator 
vectors with different support in W; {u2i,--- ,U2h',U3i, ■ ' ' ,U3g',U4ir ' ' is also a set of 

indicator vectors with different support. Define 

h 9 I 

/3i = E biUli + E + E G 

1=1 i=l j=l 
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h g I 

/32 = ^ CiU2i + ^ diUsi + ^ EjUij G M*' 
i=l 4=1 j=l 

Then we have 

h g I h g I 

i=l i=l j=l i=l 1=1 j=l 

For the matrix case, suppose g,h > 0,g + h < r, {c?j}f=i> {%}i<i<g,i<j<i are non- 

negative real numbers satisfying 

mill di > max e^, (57) 

1<«<9 

Y^tij = ej, yi<j<l (58) 

i=l 

{ci}^^i are real numbers, {usi, • • • ,u^g;u\i, ■ ■ ■ ,1*4;} is a set of orthogonal unit vectors 
in M™", {uii,-- - and {m2i,-- - ,U2h} ore tu;o seis of orthogonal unit vectors lying in the 

perpendicular space of span{usi, • • • , na^; U41, • • • , u^i}; {^31, • • • , vsg;v4^i, ■ ■ ■ , v^} is a set of 
orthogonal unit vectors in M", {fii, • • • ,vih} and {^21, • • • ,V2h} o,Te two sets of orthogonal unit 
vectors lying in the perpendicular space of span{v^i, ■ ■ ■ , v^g] V41, ■ ■ ■ , V41}. Define 

h g I 

i=l i=l j=l 

h g I 

i=l i=l j=l 

Then we have 

h g I h g I 

1=1 i=l j=l 1=1 i=l i=l 

(59) 

Proof of Lemma 15.21 We prove the Lemma by induction on I. 

When / = 0, (j59p is clear to hold by the definition of and the fact that g + h < r. Suppose 
([5911 holds for /-!,(/> 1), we note 

Yi = -U3iv'^i + u^ivJi, l<i<g (60) 
a 

P,=X,-YtiiYi, z = l,2 (61) 
1=1 
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Qiz = X,-Y^ t^iY^ + {til + di)Yi z = 1, 2, 1 < i < 



(62) 



w=l 



We can show the following equality in Z2-space: 

a 9 9 



1=1 



4 = 1 



w=l 



(63) 



i=l 



«)=1 



where z = l,2,i^, = /i = 1 - Ef=i By dST]), ii we have 



di+tii 
9 



Thus, fj,^ are all non-negative numbers satisfying fx + Yli=i = 1- Consider the difference of 
these two equalities (f63]l (z = 1, 2), we get 

\\M{X,)\\l-\\M{X2)\\l = ^ [\\M{Pr)\\l - \\M{P2)\\l] [\\M{Qa)\\l - \\M{Qi2)\\l] (64) 



i=l 



By computing directly we can get 

h g 



l-i 



i=l 
h 



i=l 
9 



«-i 



i=l 



i=l 



<3il = E b^uiwvTw + 



■u;=l 



E (d^ + t^z)'U3^?^L + (di + tii)u4ivJi 



/-I 



Qi2 = E C^^,n2u,^'|„ + 



ui=l 



/-I 



E (dt« + iu,«)^3t«^^L + (.di + tii)u4ivJi 

which corresponds with the assumption of / — 1. Now by induction assumption of Z — 1, for all 
1 < w < g we have 

\\M{Pi)\\l - \\MiP2)g > (1 - 6^){Y^ h\ + Y.{d, + - (1 + d^){Y c\ + Y^d. + E*^:'-)') 

1=1 j=l j = l j=l i=l j = l 

h g I h g I 

ii-M(Q«,i)iii - ii-M(Q«,2)iii > (1 - 5.^)(E^' + E(^* + E%)') - (1 + '^r^)(E^' + E(^^ + E*^^-)') 

i=l i=l i=l *=1 i=l 7=1 

(65) 



Together (j65|) with (|M|) . we can get (j59j) for the case /. □ 
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Lemma 5.3 Suppose m > r, ai > 02 > • • • > am > 0, X]I=i — Sl^r+i '^i' then for all a > 1, 

m r 

E«i<E«"- (66) 

j=r+l i=l 

More generally, suppose ai > 02 > • • • > Om > 0, X > and Yll=i + ^ X^i^r+i then for 
all a > 1, 

j=r+l \ / 



Proof of Lemma 15.31 It is sufficient to show the general part only. Since we can set aj = 
when j > m, we assume m > 2r without loss of generality. By Lemma 13.11 we can find 
{sij}i<i<r,2r+i<j<m Satisfying dMD, (gS]). Hence, 

m m r 2r r / m 

E = E «r(E^^.)+ E «i=E «"+^+ E "i"'^. 

j=r+l j=2r+l i=l j=r+l i=l \ j=2r+l 

r / m \ r I m. 

< E "'r+l <^r+i + E ^ E "'•+^ + E 

i=l \ jr=2r+l J i=l \ j=2r+l 

a 



r r J \Vr r 



Lemma 5.4 Suppose fi = argmin^ i/ien satisfies \\A^ A{I3 — < A/2. 

This is the vector version of Lemma 3.5 in [9], for which we omit the proof here. 
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